The present invention relates to methods and systems for encoding digital multimedia data for transmission over a network.
The following U.S. Patents have been found in a U.S. Patent Search and are believed to be generally relevant to the field of the invention:
When using various media such as video, audio, text and images, a user generally retrieves the media from a storage device or xe2x80x9cserverxe2x80x9d connected via a network to many computers or users. The server downloads the media to the network and transmits it to the user at the user""s request.
There are two basic limitations involved in such data retrieval: delay between the time that a user requests the data and the time when the server downloads it to the network, and bandwidth limitations on data throughput and rate of data transmission. The present invention relates to the second limitation.
One example of such a system includes a CD ROM drive and personal computer which may be located at the same site. Another example includes a network connecting Internet servers and users"" personal computers. Such networks are installed in order to facilitate convenient data transmission between users and data distribution from the server to the users"" computers.
When a user retrieves the media from a storage device or server, typically connected via a network, the aforesaid bandwidth limitations affect the amount of time required to transmit a video frame from the server to the user, and thus limit the video frame rate. Moreover, when dealing with object movies and panoramas the files being transmitted are extremely large, so that overcoming bandwidth limitations is a critical enabling factor, even for high bandwidth networks.
Currently, two methods are employed to overcome bandwidth limitations: The first is to compress the video frame sequence, thereby speeding up transmission time at the cost of additional downstream processing to decompress the frames prior to display. The second is to copy the entire sequence to an intermediate storage device, such as a user""s hard disk, to which the user has higher bandwidth access, at the cost of delaying the viewing of the video until the entire sequence has been delivered.
Known network applications involve streaming data from a server to a client computer (hereinafter also referred to as xe2x80x9cclientxe2x80x9d). xe2x80x9cStreamingxe2x80x9d refers to serial or parallel transmission of digital data between two computers, by transmitting sequences of bit packets. For example, installation executables on a network server stream files to a client computer performing the installation. Servers with large amounts of memory are used to archive digital movies, which are streamed to a client computer for viewing upon demand. Digital video is broadcast from cable stations to subscribers using streaming. Internet browsers, such as Netscape and Microsoft Explorer, are used to stream data from a server on the web to a client. Internet web sites can contain enormous databases, such as phone directories for all of the cities in the U.S., photographs from art galleries and museums around the world, voluminous encyclopedias, and even copies of all patents ever issued by the U.S. Patent and Trademark Office. Clients using the Internet can search these databases and then request the server to download specific information. This request initiates a streaming event.
In view of the multitude of bandwidths present in complex client/server systems today and the large amounts of data necessary to produce compelling audio, video and imaging presentations, there is a pressing demand for scalable data representation of multimedia data, so that it can be delivered for on-line interactive playback in such a form that the transmission rate can accommodate the client bandwidth. The Internet is an example of a complex system where clients with many different bandwidth constraints demand multimedia data from server sites. There are several low bandwidth rates for modem transmission over phone lines, higher rates for frame relay lines, higher rates for ISDN lines, even higher rates for T1 lines, etc.
The present invention seeks to provide an improved method and system for transmitting digital data representing the original over plural transmission links at least some of which have limited bandwidth.
The present invention relates to scalable encoding, which enables two or more clients, connected to a server by lines having differing bandwidth, to begin playing the multimedia data on-line, and both at the same time, almost immediately after the start of streaming, but the lower bandwidth client receives lower quality media than the higher bandwidth client at first. As the media is replayed in the foreground and the bandwidth is freed, more data streams in via background, and the quality of the media is enhanced.
Without scalable encoding, a client of an Internet application must wait until the requested data arrives, at whatever rate its network line provides. A client with a 14.4 Kbs modem line, for example, would have to wait twice as long as a client with a 28.8 Kbs modem line. Moreover, if the data was encoded for 28.8 Kbs on-line playback, the 14.4 Kbs client would never be able to achieve live playback, since there would be an ever-increasing lag in the data stream. Conversely, if the data were encoded for 14.4 Kbs on-line playback, the 28.8 Kbs client would receive unnecessarily poor quality media.
On account of the progressive nature of the data representation in accordance with the present invention, the additional data block arriving in a 14.4 Kbs stream combines with the previous data block which arrived in a 14.4 Kbs stream, to produce a 28.8 Kbs streamed version; all that is being sent is the incremental data necessary for the upgrade. The progressive form of the encoding itself provides the ability to achieve scalability.
Another shortcoming of non-scalable encoding as in the prior art is the inability to preview a video sequence. Often a client would like to play a quick preview of a video clip, before deciding whether or not to download it. The scalable representation of the present invention can be used to deliver the video in a preview mode, as the first data blocks. If the client continues to download the video after previewing, the first data block already transmitted is progressively integrated with additional data blocks to create the full viewing video.
The present invention can also be applied to enhance delivery of large still images for multi-resolution gazing. Current technology transmits such images as large files, and carries out extensive computations for sub-sampling to lower resolution and zooming in to areas of interest for gazing. This makes it very time consuming to interact with large images, and as a result it is currently impractical to produce high resolution images for Internet browsing. When dealing with large images, producers simply sub-sample them to fit entirely within a computer monitor screen, and store the resulting low resolution images on web servers. Using the technology of the present invention, producers can deliver high resolution images over the Internet for rapid interactive gazing.
The present invention seeks to provide a scalable representation of multimedia data, enabling the data to be (a) progressively streamed, (b) transmitted asynchronously to clients at different bandwidths and (c) played back interactively on-line. The representation is two-dimensional, with one dimension (block number) being characterized by progressiveness in quality, and the second dimension (frame number) being characterized by interactivity.
The representation comprises data blocks which are integrated with one another to produce successively higher bandwidth versions of the media, the data blocks comprising encoded frames. The first data block corresponds to the lowest bandwidth, and enables the client with this bandwidth to play back the media on-line at the lowest quality. The second data block, when integrated with the first block, corresponds to the next higher bandwidth, and enables the client with this bandwidth to play back the media on-line at the next highest quality, and similarly for each successive data block.
Moreover, a client with the lowest bandwidth who played the media at the low quality and freed the bandwidth can continue in background to receive successive data blocks and integrate them with previously received data blocks, resulting in successively higher quality media each time it is replayed. The modular form of the data representation thus makes it possible to both accommodate different bandwidths and progressively update media quality. A production tool makes it possible for a producer to control modularity and quality settings.
In accordance with the present invention there is provided a method for providing on-line virtual reality movies, comprising inputting a cyclic movie sequence into an encoder, determining the number of portions that each frame of said movie is divided into, and forming partial frames, specifying hot-spots and independent objects for interaction within a partial frame, transmitting the partial frames part by part to a user""s asynchronous database, and displaying said frames on a user""s interface.
There is also provided in accordance with a preferred embodiment of the present invention a system for producing virtual reality (VR) movies comprising an encoder for preparing the VR movie for transmission, and a server including a repository for the VR movie and a transceiver for transmitting the movie, part by part to a user, upon request.
An essential feature of the present invention is the use of a two-dimensional interactive progressive database to represent multimedia data, and the storage of this database in three different forms: for streaming, processing and playback purposes. The database is calibrated in data blocks of roughly equal size, to deliver the media for on-line playback at a selected range of bandwidths, and in such a way that the higher bandwidth versions are built by integrating data blocks with the lower bandwidth versions. Thus, rather than discard the lower bandwidth data, it is saved and used directly to upgrade from low to high bandwidth quality.
The data blocks themselves are comprised of frames which can be randomly accessed, thus giving a second dimension (namely, frames) to the progressive database. Thus it is possible to selectively build higher quality versions of some frames and not others. The mechanism determining which frames to send within each block may be controlled interactively by the user.
To best appreciate the achievement of the present invention, one should consider it in the broader perspective of streaming technology. There are known two modes of accessing streamed data: random access and sequential access. In a random access mode, the client can request the server to transmit data from any position within the stream. In a sequential access mode, the server can only send a serial stream.
Current Internet streaming technology does not provide random access. Rather, the stream is sent serially in packets. The present invention operates by creating three copies of the progressive database. A first copy is stored on the server serially. A second copy, which mirrors the server database, is built on the client, with random accessibility. These first two copies are in encoded form. The frames in the data blocks from the second copy are decoded and stored in the third copy. The third copy is dynamically updated and contains the frames to be displayed in either raw bitmap form, or in intermediate compressed form whereby the decompression is fast enough to keep up with real time interactive display in response to user commands.
A by-product of the present invention, when applied to video clips, is the ability to deliver a preview of the video using the first data blocks. This enables the client to play the preview almost immediately after the transmission begins, and then to quickly decide whether or not to proceed with the download. Moreover, if the client does continue with the download, then the first data block already transmitted is integrated with additional data blocks being downloaded, to form the full view version of the video. Thus, rather than discard the data transmitted for the preview, it is saved and used to create the full view frame sequence.
The present invention can also be applied to efficiently deliver large still images at multiple resolutions for interactive gazing. Each block of the progressive database stores various tiles of the image at different resolutions. Smaller tiles are stored at higher resolution. Hot spots are used to link tiles at lower resolution to smaller tiles contained within them at higher resolution. When a viewer clicks on a hot spot to gaze, the display quickly brings up the tile linked to by the hot spot, giving the effect of an instant zoom in. The totality of multi-resolution tiles may comprise the xe2x80x9cframesxe2x80x9d in this application, and these frames form the interactivity dimension of the database.
The first tile consists of the lowest resolution version of the full image, with hot spots encoded within it. As soon as the first tile is delivered and decoded, the user can at once begin gazing at the higher resolution tiles, even though the image is of low quality. While the user is viewing the tiles, additional data blocks are being delivered and decoded in background, and the quality of the tiles is being upgraded as time progresses. In other words, the zoomed in portion of the image being gazed at gets displayed almost immediately after the streaming begins, but at low quality. The quality improves with time.
The higher resolution tiles correspond to hierarchical xe2x80x9careas of interestxe2x80x9d in the image. The choice of which areas to mark with hot spots as areas of interest is in the hands of the producer. This use of the invention is particularly efficient in the case where there is a relatively small number of areas of interest in the full image, so that relatively few tiles are encoded at the higher resolutions. Without the present invention, the client would have to wait to receive the full image at higher resolution before viewing any part of it at this resolution, even though only small parts of it are of interest. Moreover, each zoom in and out would be both processor and memory demanding. Viewer interactivity would be painfully slow.
There is thus provided in accordance with a preferred embodiment of the present invention a system for transmitting digital data representing the original over plural transmission links at least some of which have limited bandwidth including:
a digital data source storing digital data representing an original;
a digital data receiver receiving the digital data representing an original via one of the plural transmission links having limited bandwidth; and
a digital data transmitter operative to transmit the digital data representing an original to the receiver over a transmission link having a limited bandwidth in plural blocks which are sequentially transmitted at a rate determined by the limited bandwidth, each block being an incomplete collection of data which includes parts of multiple frames, each frame being viewable in a selectable order by the receiver even when less than all of the plural blocks have been received, receipt of subsequent blocks by the receiver being used to cumulatively improve the quality of the digital data viewed by the receiver.
There is additionally provided in accordance with a preferred embodiment of the present invention a digital data transmitter actuator comprising:
an organizer operative, when actuated, to access digital data representing an original which is organized in plural blocks for subsequent transmission, each block being an incomplete collection of data which includes parts of multiple frames, each frame being viewable in a selectable order by the a receiver even when less than all of the plural blocks have been received; and
a receiver instruction interface responsive to interactive inputs from a receiver for actuating the organizer to select a given block and at least one given partial frame within the given block for transmission.
There is also provided in accordance with a preferred embodiment of the present invention a digital data receiver including:
a data receipt interface receiving digital data representing an original in a plurality of sequential blocks, each block being an incomplete collection of data which includes parts of multiple frames;
a block accumulator for combining plural blocks as they are received for viewing by the recipient; and
a viewer including a recipient interface which permits each frame to be viewed in an order selected by the recipient, even when less than all of the plural blocks have been received, combining of plural blocks by the block accumulator being used to improve the quality of the digital data viewed by the recipient.
There is additionally provided in accordance with a preferred embodiment of the present invention a method for transmitting digital data representing an original over plural transmission links at least some of which have limited bandwidth including the steps of:
storing digital data representing an original;
receiving at a receiver the digital data representing an original via one of the plural transmission links having limited bandwidth; and
transmitting the digital data representing an original to the receiver over a transmission link having a limited bandwidth in plural blocks which are sequentially transmitted at a rate determined by the limited bandwidth, each block being an incomplete collection of data which includes parts of multiple frames, each frame being viewable in a selectable order by the receiver even when less than all of the plural blocks have been received, receipt of subsequent blocks by the receiver being used to cumulatively improve the quality of the digital data viewed by the receiver.
There is also provided in accordance with a preferred embodiment of the present invention a method for digital data transmission including:
organizing digital data representing an original into plural blocks for subsequent transmission, each block being an incomplete collection of data which includes parts of multiple frames, each frame being viewable in a selectable order by the a receiver even when less than all of the plural blocks have been received;
responsive to interactive inputs from a receiver for actuating the organizer, selecting a given block and at least one given partial frame within the given block for transmission; and
transmitting the selected given block and at least one given partial frame to a user.
In accordance with a preferred embodiment of the present invention, the block accumulator is operative to combine plural blocks which are distinguished from each other by their respective frequency bands.
Preferably, the digital data receiver includes a fractal decompression engine.
In accordance with a preferred embodiment of the present invention, the data receipt interface is operative to initially receive a first plurality of blocks containing relatively low frequency data and thereafter receive a second plurality of blocks containing relatively high frequency data and the block accumulator is operative to reconstitute the digital data representing an original from the blocks representing relatively high frequency and relatively low frequency data.
Preferably, the block accumulator is operative to combine plural blocks having different sampling.
In accordance with a preferred embodiment of the present invention, the sampling rate of a combined plurality of blocks is equal to the sum of the sampling rates of individual ones of the plurality of blocks.
Preferably, the digital data receiver includes a wavelet decoder.
In accordance with a preferred embodiment of the present invention, the block accumulator includes a dequantizer which combines blocks each of which contain quantized data of a different order, such that accumulation of multiple blocks provides combined data of greater precision than that contained in any single block.
There is additionally provided in accordance with a preferred embodiment of the present invention a digital data transmitter actuator including:
an organizer operative, when actuated, to access digital data representing an original which is organized in plural blocks for subsequent transmission, each block being an incomplete collection of data which includes parts of multiple frames, each frame being viewable in a selectable order by the a receiver even when less than all of the plural blocks have been received; and
a receiver instruction interface responsive to interactive inputs from a receiver for actuating the organizer to select a given block and at least one given partial frame within the given block for transmission.
In accordance with a preferred embodiment of the present invention a first one of the plural blocks contains digital data which represents a first approximation to the original.
Preferably, additional ones of the plural blocks, when combined with the first one of the plural blocks provide additionally accurate approximations to the original.
In accordance with a preferred embodiment of the present invention each of the multiple frames includes a portion of data which can be independently and interactively manipulated.
Preferably, the system also includes a block generator operative to receive digital data representing the original and to provide the plural blocks.
There is also provided in accordance with a preferred embodiment of the present invention a block generator including:
a producer interface; and
a digital data compressor, operative in response to producer control parameters received via the producer interface for receiving digital data representing an original and providing plural blocks, each block being an incomplete collection of data which includes parts of multiple frames.
In accordance with a preferred embodiment of the present invention the block generator is operative to provide plural blocks which are distinguished from each other by their respective frequency bands.
Preferably, the block generator includes a fractal compression engine.
In accordance with a preferred embodiment of the present invention, the block generator is operative to decompose the digital data representing an original in relatively high frequency and relatively low frequency digital data portions, and wherein a first plurality of blocks containing the relatively low frequency portion is transmitted by the data transmitter prior to transmission of a second plurality of blocks containing the relatively high frequency portion.
Preferably, the block generator is operative to provide plural blocks by sampling the received digital data.
In accordance with a preferred embodiment of the present invention the sampling rate of a plurality of blocks is equal to the sum of the sampling rates of individual ones of the plurality of blocks.
Preferably, the block generator includes a wavelet encoder.
In accordance with a preferred embodiment of the present invention, the block generator includes a quantizer which produces blocks each of which contain quantized data of a different order, such that accumulation of multiple blocks provides combined data of greater precision than that contained in any single block.
There is also provided in accordance with another preferred embodiment of the present invention, a method for encoding original digital video data to be stored on a server computer for on-line delivery to client computers, including the steps of:
encoding the digital video into a database including a series of encoded data blocks, each block including a sequence of encoded frames, with the property that successive blocks when decoded and integrated together provide successively higher bandwidth versions of the video for on-line playback;
storing the database on a server computer;
processing a request by a client computer for on-line delivery of the video in order to determine which data blocks to transmit, so as to accommodate the client bandwidth;
transmitting the necessary data blocks to the client;
decoding the data blocks on the client computer;
integrating the data blocks together on the client computer to reconstruct an appropriate version of the original digital video; and
playing the reconstructed video on the client computer.
Further in accordance with a preferred embodiment of the present invention, the step of encoding includes a bit-rate control device enabling the producer to pre-select the sequence of bandwidths or quality levels for the database.
Still further in accordance with a preferred embodiment of the present invention, the step of encoding is performed in such a way that the first blocks of the database correspond to previews of the video.
Additionally in accordance with a preferred embodiment of the present invention, the steps of transmitting, decoding, integrating and playing are repeated in succession a number of times in order to transmit additional data blocks to the client, thereby upgrading the quality of the video while it is replayed.
There is also provided in accordance with another preferred embodiment of the present invention a method for encoding original digital audio data to be stored on a server computer for on-line delivery to client computers, including the steps of:
encoding the digital audio into a database including a series of encoded data blocks, each block including a sequence of encoded frames, with the property that successive blocks when decoded and integrated together provide successively higher bandwidth versions of the audio for on-line playback;
storing the database on a server computer;
processing a request by a client computer for on-line delivery of the audio in order to determine which data blocks to transmit, so as to accommodate the client bandwidth;
transmitting the necessary data blocks to the client;
decoding the data blocks on the client computer;
integrating the data blocks together on the client computer to reconstruct an appropriate version of the original digital audio; and
playing the reconstructed audio on the client computer.
Further in accordance with a preferred embodiment of the present invention, the step of encoding includes a bit-rate control device enabling the producer to pre-select the sequence of bandwidths or quality levels for the database.
Still further in accordance with a preferred embodiment of the present invention, the steps of transmitting, decoding, integrating and playing are repeated in succession a number of times in order to transmit additional data blocks to the client, thereby upgrading the quality of the audio while it is replayed.
There is also provided in accordance with another preferred embodiment of the present invention, a method for encoding original digital object movie data to be stored on a server computer for on-line delivery to client including, comprising the steps of:
encoding the digital object movie into a database including a series of encoded data blocks, each block including a sequence of encoded frames, with the property that successive blocks when decoded and integrated together provide successively higher bandwidth versions of the object movie for on-line playback;
storing the database on a server computer;
processing a request by a client computer for on-line delivery of the object movie in order to determine which data blocks to transmit, so as to accommodate the client bandwidth;
transmitting the necessary data blocks to the client;
decoding the data blocks on the client computer;
integrating the data blocks together on the client computer to reconstruct an appropriate version of the original digital object movie; and
playing the reconstructed object movie on the client computer.
Further in accordance with a preferred embodiment of the present invention, the step of encoding includes a bit-rate control device enabling the producer to pre-select the sequence of bandwidths or quality levels for the database.
Still further in accordance with a preferred embodiment of the present invention, the steps of transmitting, decoding, integrating and playing are repeated in succession a number of times in order to transmit additional data blocks to the client, thereby upgrading the quality of the object movie while it is replayed.
There is also provided in accordance with another preferred embodiment of the present invention a method for encoding an original digital panorama to be stored on a server computer for on-line delivery to client computers, including the steps of:
encoding the digital panorama into a database including a series of encoded data blocks, each block including a sequence of encoded frames, with the property that successive blocks when decoded and integrated together provide successively higher bandwidth versions of the panorama for on-line playback;
storing the database on a server computer;
processing a request by a client computer for on-line delivery of the panorama in order to determine which data blocks to transmit, so as to accommodate the client bandwidth;
transmitting the necessary data blocks to the client;
decoding the data blocks on the client computer;
integrating the data blocks together on the client computer to reconstruct an appropriate version of the original digital panorama; and
playing the reconstructed panorama on the client computer.
Further in accordance with a preferred embodiment of the present invention the step of encoding includes a bit-rate control device enabling the producer to pre-select the sequence of bandwidths or quality levels for the database.
Still further in accordance with a preferred embodiment of the present invention the steps of transmitting, decoding, integrating and playing are repeated in succession a number of times in order to transmit additional data blocks to the client, thereby upgrading the quality of the panorama while it is replayed.
There is also provided in accordance with another preferred embodiment of the present invention a method for encoding original digital large still image data to be stored on a server computer for on-line delivery to client computers, including the steps of:
encoding the large digital image into a database including a series of encoded data blocks, each block including a sequence of encoded multi-resolution tiles of the image, with the property that successive blocks when decoded and integrated together provide successively higher quality versions of the tiles for display;
storing the database on a server computer;
processing a request by a client computer for on-line delivery of the image in order to determine which data blocks to transmit;
transmitting the necessary data blocks to the client;
decoding the data blocks on the client computer;
integrating the data blocks together on the client computer to reconstruct an appropriate version of the original multi-resolution image tiles; and
interactively displaying the reconstructed tiles on the client computer.
Still further in accordance with a preferred embodiment of the present invention the step of encoding includes a compression control device enabling the producer to pre-select the sequence of quality levels for the database.
Additionally in accordance with a preferred embodiment of the present invention the step of encoding operates on a plurality of images forming an animation, and each encoded data block is comprised of multi-resolution tiles from the plurality of images.
There is also provided in accordance with another preferred embodiment of the present invention a video processing system operative on digital video data for encoding the digital video, storing it on a server computer and delivering it to client computers on-line upon request including:
an encoder for compressing the digital video into a database including a series of encoded data blocks, each block including a sequence of encoded frames, with the property that successive blocks when decoded and integrated together provide successively higher bandwidth versions of the video for on-line playback;
a storage device for archiving the database on a server computer;
a processing unit for accepting a request by a client computer for on-line delivery of the video and determining which data blocks to transmit, so as to accommodate the client bandwidth;
a transmitter for delivering the necessary data blocks to the client;
a decoder for decompressing the data blocks back into video data on the client computer;
an accumulator for integrating the data blocks together on the client computer to reconstruct an appropriate version of the original digital video; and
a player on the client computer for playing the reconstructed digital video.
Additionally in accordance with a preferred embodiment of the present invention the encoder includes a bit-rate controller enabling the user to pre-select the sequence of bandwidths or quality levels for the database.
Moreover in accordance with a preferred embodiment of the present invention the encoder compresses the digital video in such a way that the first blocks of the database correspond to previews of the video.
Further in accordance with a preferred embodiment of the present invention the transmitter, decoder, accumulator and player repeatedly operate in succession a number of times in order to transmit additional data blocks to the client, thereby upgrading the quality of the video while it is being replayed.
There is also provided in accordance with another preferred embodiment of the present invention an audio processing system operative on digital audio data for encoding the digital audio, storing it on a server computer and delivering it to client computers on-line upon request including:
an encoder for compressing the digital audio into a database including a series of encoded data blocks, each block including a sequence of encoded frames, with the property that successive blocks when decoded and integrated together provide successively higher bandwidth versions of the audio for on-line playback;
a storage device for archiving the database on a server computer;
a processing unit for accepting a request by a client computer for on-line delivery of the audio and determining which data blocks to transmit, so as to accommodate the client bandwidth;
a transmitter for delivering the necessary data blocks to the client;
a decoder for decompressing the data blocks back into audio data on the client computer;
an accumulator for integrating the data blocks together on the client computer to reconstruct an appropriate version of the original digital audio; and
a player on the client computer for playing the reconstructed digital audio.
Further in accordance with a preferred embodiment of the present invention the encoder includes a bit-rate controller enabling the user to pre-select the sequence of bandwidths or quality levels for the database.
Still further in accordance with a preferred embodiment of the present invention the transmitter, decoder, accumulator and player repeatedly operate in succession a number of times in order to transmit additional data blocks to the client, thereby upgrading the quality of the audio while it is being replayed.
There is also provided in accordance with another preferred embodiment of the present invention an object movie processing system operative on digital object movie data for encoding the digital object movie, storing it on a server computer and delivering it to client computers on-line upon request including:
an encoder for compressing the digital object movie into a database including a series of encoded data blocks, each block comprising a sequence of encoded frames, with the property that successive blocks when decoded and integrated together provide successively higher bandwidth versions of the object movie for on-line playback;
a storage device for archiving the database on a server computer;
a processing unit for accepting a request by a client computer for on-line delivery of the object movie and determining which data blocks to transmit, so as to accommodate the client bandwidth;
a transmitter for delivering the necessary data blocks to the client;
a decoder for decompressing the data blocks back into object movie data on the client computer;
an accumulator for integrating the data blocks together on the client computer to reconstruct an appropriate version of the original digital object movie; and
a player on the client computer for playing the reconstructed digital object movie.
Further in accordance with a preferred embodiment of the present invention the encoder includes a bit-rate controller enabling the user to pre-select the sequence of bandwidths or quality levels for the database.
Still further in accordance with another preferred embodiment of the present invention the transmitter, decoder, accumulator and player repeatedly operate in succession a number of times in order to transmit additional data blocks to the client, thereby upgrading the quality of the object movie while it is being replayed.
There is also provided in accordance with another preferred embodiment of the present invention a panorama processing system operative on digital panorama data for encoding the digital panorama, storing it on a server computer and delivering it to client computers on-line upon request including:
an encoder for compressing the digital panorama into a database including a series of encoded data blocks, each block including a sequence of encoded frames, with the property that successive blocks when decoded and integrated together provide successively higher bandwidth versions of the panorama for on-line playback;
a storage device for archiving the database on a server computer;
a processing unit for accepting a request by a client computer for on-line delivery of the panorama and determining which data blocks to transmit, so as to accommodate the client bandwidth;
a transmitter for delivering the necessary data blocks to the client;
a decoder for decompressing the data blocks back into panorama data on the client computer;
an accumulator for integrating the data blocks together on the client computer to reconstruct an appropriate version of the original digital panorama; and
a player on the client computer for playing the reconstructed digital panorama.
Further in accordance with a preferred embodiment of the present invention the encoder includes a bit-rate controller enabling the user to pre-select the sequence of bandwidths or quality levels for the database.
Still further in accordance with a preferred embodiment of the present invention the transmitter, decoder, accumulator and player repeatedly operate in succession a number of times in order to transmit additional data blocks to the client, thereby upgrading the quality of the panorama while it is being replayed.
There is also provided in accordance with another preferred embodiment of the present invention an image processing system operative on large digital image data for encoding the digital image, storing it on a server computer and delivering it to client computers on-line upon request including:
an encoder for compressing the large digital image into a database including a series of encoded data blocks, each block including a sequence of encoded multi-resolution tiles of the image, with the property that successive blocks when decoded and integrated together provide successively higher quality versions of the image tiles;
a storage device for archiving the database on a server computer;
a processing unit for accepting a request by a client computer for on-line delivery of the image and determining which data blocks to transmit;
a transmitter for delivering the necessary data blocks to the client;
a decoder for decompressing the data blocks back into image tile data on the client computer;
an accumulator for integrating the data blocks together on the client computer to reconstruct an appropriate version of the original multi-resolution image tiles; and
an interactive viewer on the client computer for displaying the reconstructed image tiles.
Additionally in accordance with a preferred embodiment of the present invention the encoder includes a compression controller enabling the user to pre-select the sequence of quality levels for the database.
Moreover in accordance with a preferred embodiment the encoder operates on a plurality of images forming an animation, and each encoded data block is comprised of multi-resolution tiles from the plurality of images.
There is also provided in accordance with another preferred embodiment of the present invention a method for caching of data which gets transmitted from servers to clients on a central hub within a network, including the steps of:
encoding digital multimedia data into databases including a series of encoded data blocks, each block including a sequence of encoded frames, with the property that successive blocks when decoded and integrated together provide successively higher bandwidth versions of the media for on-line playback;
storing the databases on a multitude of server computers;
managing within the hub requests by client computers for on-line delivery of media stored on server computers in order to determine which data blocks to transmit, so as to accommodate the client bandwidth;
transmitting the necessary data blocks from the server and from the hub to the client;
storing the data blocks delivered by the server in the cache residing in the central hub;
processing within the hub the data blocks it receives;
decoding the data blocks on the client computer;
integrating the data blocks together on the client computer to reconstruct an appropriate version of the original digital media; and
playing the reconstructed media on the client computer.
Further in accordance with a preferred embodiment of the present invention the step of managing is performed by:
setting inventory flags to indicate which data blocks are currently stored in the hub.
Still further in accordance with a preferred embodiment of the present invention the step of managing further including the steps of:
communicating with the servers to monitor which media data is outdated;
removing from cache the blocks corresponding to the media data which is outdated; and
resetting the inventory flags to indicate that the above blocks are no longer stored in the cache.
Still further in accordance with a preferred embodiment of the present invention the step of processing comprising the steps of:
decoding the data blocks received;
integrating the data blocks together to reconstruct appropriate versions of the original digital media; and
encoding the reconstructed media versions into an intermediate database for future transmission to the clients.
There is also provided in accordance with another preferred embodiment of the present invention a proxy system operative on a server/client network for caching of data which gets transmitted from servers to clients on a central hub, including:
an encoder for compressing digital multimedia data into databases including a series of encoded data blocks, each block including a sequence of encoded frames, with the property that successive blocks when decoded and integrated together provide successively higher bandwidth versions of the media for on-line playback;
server communication lines from the servers to the hub for sending data blocks;
client communication lines from the hub to the clients for sending digital data;
storage devices for archiving the databases on a multitude of server computers;
a management unit within the hub to process requests by client computers for on-line delivery of media stored on server computers in order to determine which data blocks to transmit, so as to accommodate the client bandwidth;
a transmitter for delivering the necessary data blocks on the server communication lines from the server to the hub, and on the client communication lines from the hub to the client;
a storage device for saving the data blocks delivered by the server communication lines in the cache residing in the central hub;
a processing unit within the hub for processing the data blocks which the hub receives;
a decoder for decompressing the data blocks on the client computer;
an accumulator for integrating the data blocks together on the client computer to reconstruct an appropriate version of the original digital media; and
a player for playing the reconstructed media on the client computer.
Further in accordance with a preferred embodiment of the present invention the management unit operates by setting inventory flags to indicate which data blocks are currently stored in the hub.
Still further in accordance with a preferred embodiment of the present invention the management unit operates by monitoring from the servers which media data is outdated, removing from cache the blocks corresponding to the media data which is outdated, and resetting the inventory flags to indicate that the above blocks are no longer stored in the cache.
Additionally in accordance with a preferred embodiment of the present invention the processing unit comprising:
a decoder for decompressing the data blocks received;
an accumulator for integrating the data blocks together to reconstruct appropriate versions of the original digital media; and
an encoder for compressing the reconstructed media versions into an intermediate database for future transmission to the clients.
There is also provided in accordance with another preferred embodiment of the present invention a multi-casting unit (MCU) system operative on a broadcasting network for caching of data which gets transmitted from stations to viewers, including:
an encoder for compressing digital multimedia data into databases including a series of encoded data blocks, each block including a sequence of encoded frames, with the property that successive blocks when decoded and integrated together provide successively higher bandwidth versions of the media for on-line playback;
station communication lines from the stations to the MCU for sending data blocks;
viewer communication lines from the MCU to the viewers for sending data;
viewer receiver units for receiving the data sent by the MCU;
storage devices for archiving the databases on a multitude of station computers;
a management unit within the MCU to process requests by viewers for on-line delivery of media stored on station computers in order to determine which data blocks to transmit, so as to accommodate the viewer bandwidth;
a transmitter for delivering the necessary data blocks on the station communication lines from the station to the MCU, and on the viewer communication lines from the MCU to the viewer receiver units;
a storage device for saving the data blocks delivered by the station communication lines in the cache residing in the MCU;
a processing unit within the MCU for processing the data blocks which the MCU receives;
a decoder for decompressing the data blocks on the viewer receiver;
an accumulator for integrating the data blocks together on the viewer receiver unit to reconstruct an appropriate version of the original digital media; and
a player for playing the reconstructed media from the viewer receiver unit.
Further in accordance with a preferred embodiment of the present invention the management unit operates by setting inventory flags to indicate which data blocks are currently stored in the MCU.
Still further in accordance with a preferred embodiment of the present invention the management unit operates by monitoring from the stations which media data is outdated, removing from cache the blocks corresponding to the media data which is outdated, and resetting the inventory flags to indicate that the above blocks are no longer stored in the cache.
Additionally in accordance with a preferred embodiment of the present invention the processing unit within the MCU including:
a decoder for decompressing the data blocks received;
an accumulator for integrating the data blocks together to reconstruct appropriate versions of the original digital media; and
an encoder for compressing the reconstructed media versions into an intermediate database for future transmission to the viewers.
There is also provided in accordance with another preferred embodiment of the present invention a method for streaming multimedia data over a network, including the steps of:
encoding the media into a progressive database indexed according to frame and progressive block numbers;
serializing the encoded database;
storing the serialized database on a server;
streaming the serialized database to a client upon request;
creating a mirror copy of the encoded database on the client computer from the data which streams in; and
decoding the encoded database on the client computer into a sequence of frames for real time display.
There is also provided in accordance with another preferred embodiment of the present invention a multimedia network streaming system, including:
an encoder for compressing the media into a progressive database indexed according to frame and progressive block numbers;
a sequencer for serializing the encoded database;
a storage device for archiving the serialized database on a server;
a transmitter for streaming the serialized database to a client upon request;
a processor for creating a mirror copy of the encoded database on the client computer from the data which streams in; and
a decoder for decompressing the encoded database on the client computer into a sequence of frames for real time display.
There is also provided in accordance with another preferred embodiment of the present invention a system for transmitting model based data representations of three dimensional images over plural transmission links having limited bandwidth, said system including:
a digital data source storing model based data representations of three dimensional images;
an image processor for rendering views of said model based data representations into raster bitmap format;
a digital data receiver receiving said digital data in said raster bitmap format over a one of the plural transmission links having limited bandwidth; and
a digital data transmitter operative to transmit the digital data in said raster bitmap format to said receiver over a transmission link having a limited bandwidth in plural blocks which are sequentially transmitted at a rate determined by the limited bandwidth, each block being an incomplete collection of data which includes parts of multiple frames, each frame being viewable in a selectable order by said receiver even when less than all of the plural blocks have been received, receipt of subsequent blocks by the receiver being used to cumulatively improve the quality of the digital data viewed by the receiver.
Further in accordance with a preferred embodiment of the present invention the model based data representations comprise VRML representations.
Still further in accordance with a preferred embodiment of the present invention the model based data representations comprise CAD-CAM representations.
Additionally in accordance with a preferred embodiment of the present invention the image processor is operative to render only views which are selected by a user.
There is also provided in accordance with another preferred embodiment of the present invention a method for transmitting model based data representations of three dimensional images over plural transmission links having limited bandwidth, said system including:
storing model based data representations of three dimensional images;
rendering views of the model based data representations into raster bitmap format;
receiving the digital data in said raster bitmap format over a one of said plural transmission links having limited bandwidth; and
transmit the digital data in said raster bitmap format to the receiver over a transmission link having a limited bandwidth in plural blocks which are sequentially transmitted at a rate determined by the limited bandwidth, each block being an incomplete collection of data which includes parts of multiple frames, each frame being viewable in a selectable order by the receiver even when less than all of the plural blocks have been received, receipt of subsequent blocks by the receiver being used to cumulatively improve the quality of the digital data viewed by the receiver.
Further in accordance with a preferred embodiment of the present invention the model based data representations comprise VRML representations, and CAD-CAM representations.
Still further in accordance with a preferred embodiment of the present invention the image processor is operative to render only views which are selected by a user.
The following definitions are employed throughout the specification and claims:
RESOLUTIONxe2x80x94The relationship between the number of digital samples per unit of an original and the number of digital samples per unit in a rendered version thereof Specifically, when dealing with images, resolution refers to the relationship between the number of pixels per unit area of an original image or scene and the number of pixels per unit area in a displayed image. Specifically, when dealing with audio, resolution refers to the relationship between the number of samples per unit time of an original sound and the number of samples per unit time in a played sound.
QUALITYxe2x80x94The degree to which a rendered version of an original is faithfuil to the original. Specifically, when dealing with images, quality refers to the degree to which the displayed image is faithful to the original image or scene. Normally this is expressed as the degree to which the approximation of pixel values in the displayed image approaches the correct pixel values in the original image or scene. Specifically, when dealing with audio, quality refers to the degree to which a played sound is faithful to the original sound.
FRAMExe2x80x94A portion of an original which can be independently and interactively manipulated. Specifically, when dealing with images, frame refers to a portion of an image or of a collection of images which can be independently and interactively manipulated. Specifically, when dealing with audio, frame refers to a portion of a sound which is delimited in time and can be independently and interactively manipulated.
BLOCKxe2x80x94A sequentially transmitted collection of partial data which is used to build multiple frames. The frames are built up of one or more sequentially transmitted blocks, whose contents are accumulated. Specifically, when dealing with images, the block contains image data. Specifically when dealing with audio, the block contains audio data.
PARTIAL FRAMExe2x80x94The part of a frame which is contained in a given block.
TILExe2x80x94A window sized pixel array of a predetermined given size forming part of an image. For example, tiles partition an image into a plurality of arrays, each of which contains an identical number of pixels.